Teleoperation System With Visual Indicator and Method of Use During Surgical Procedures

ABSTRACT

Teleoperation systems and methods for use during a surgical procedure. The teleoperation system comprises a surgical tool and a master input device for being moved by a user. A slave device is coupled to the surgical tool and responsive to movement of the master input device. The slave device is configured to be operated in a first mode or a second mode. In the first mode, the surgical tool is placed in a desired pose with respect to a target region in response to a manual force applied to the slave device. In the second mode, the surgical tool is manipulated during the surgical procedure in response to movement of the master input device by the user. A visual indicator is coupled to the surgical tool or the slave device to emit light in the second mode to provide visual information to the user in the second mode.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is continuation of U.S. patent application Ser. No.12/144,507, filed Jun. 23, 2008, published Jan. 1, 2009, which is adivisional of U.S. patent application Ser. No. 11/357,197, filed Feb.21, 2006, published Jun. 29, 2006, now U.S. Pat. No. 8,010,180, which isa continuation-in-part of U.S. patent application Ser. No. 10/384,072,filed Mar. 6, 2003, published Feb. 5, 2004, now U.S. Pat. No. 7,206,626;U.S. patent application Ser. No. 10/384,077, filed Mar. 6, 2003,published Feb. 19, 2004, now U.S. Pat. No. 7,206,627; U.S. patentapplication Ser. No. 10/384,078, filed Mar. 6, 2003, published Feb. 19,2004, now U.S. Pat. No. 8,095,200; and U.S. patent application Ser. No.10/384,194, filed Mar. 6, 2003, published Feb. 19, 2004, now U.S. Pat.No. 7,747,311, each of which claims priority from U.S. ProvisionalPatent Application No. 60/362,368, filed Mar. 6, 2002. U.S. patentapplication Ser. No. 11/357,197 is also a continuation-in-part of U.S.patent application Ser. No. 10/621,119, filed Jul. 16, 2003, publishedJun. 3, 2004, now U.S. Pat. No. 7,831,292, which is acontinuation-in-part of U.S. patent application Ser. No. 10/384,072,filed Mar. 6, 2003, published Feb. 5, 2004, now U.S. Pat. No. 7,206,626and U.S. patent application Ser. No. 10/384,078, filed Mar. 6, 2003,published Feb. 19, 2004, now U.S. Pat. No. 8,095,200, each of whichclaims priority from U.S. Provisional Patent Application Ser. No.60/362,368, filed Mar. 6, 2002. U.S. patent application Ser. No.11/357,197 further claims priority from U.S. Provisional PatentApplication Ser. No. 60/655,642, filed Feb. 22, 2005, and U.S.Provisional Patent Application Ser. No. 60/759,186, filed Jan. 17, 2006.Each of the above-referenced published applications and patents isincorporated by reference herein in its entirety.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to computer-assisted surgerysystems and surgical navigation systems, and more particularly to ateleoperation system with visual indicator and method of use duringsurgical procedures.

BACKGROUND OF THE INVENTION

The functions of a computer-assisted surgery (CAS) system may includepre-operative planning of a procedure, presenting pre-operativediagnostic information and images in useful formats, presenting statusinformation about a procedure as it takes place, and enhancingperformance. The CAS system may be used for procedures in traditionaloperating rooms, interventional radiology suites, mobile operating roomsor outpatient clinics. Many approaches to CAS have been attemptedcommercially. The procedure may be any medical procedure, whethersurgical or non-surgical.

Navigation systems are used to display the positions of surgical toolswith respect to pre- or intraoperative image datasets. These imagesinclude intraoperative images, such as two-dimensional fluoroscopicimages, and preoperative three dimensional images generated using, forexample, magnetic resonance imaging (MRI), computer tomography (CT) andpositron emission tomography (PET). The most popular navigation systemsmake use of a tracking or localizing system. These systems locatemarkers attached or fixed to an object, such as an instrument or apatient, and track the position of markers. These tracking systems areoptical and magnetic, but also include acoustic systems. Optical systemshave a stationary stereo camera pair that observes passive reflectivemarkers or active infrared LEDs attached to the tracked tools. Magneticsystems have a stationary field generator that emits a magnetic fieldthat is sensed by small coils integrated into the tracked tools. Thesesystems are sensitive to nearby metal objects.

While navigation systems are relatively easy to integrate into theoperating room, a fundamental limitation is that they have restrictedmeans of communication with the surgeon. Most systems transmitinformation to the surgeon via a computer monitor. Conversely, thesurgeon transmits information to the system via a keyboard and mouse,touchscreen, voice commands, control pendant, or foot pedals, and alsoby moving the tracked tool. The visual displays of navigation systemsmay at best display multiple slices through three-dimensional diagnosticimage datasets, which are not easy to interpret for complex 3-Dgeometries. These displays also require the surgeon to focus his visualattention away from the surgical field.

When defining a plan using a tracked tool, it can be difficult tosimultaneously position the tool appropriately in multiple degrees offreedom (DOFs). Similarly, when aligning a tracked instrument with aplan, it is difficult to control the position of the tool in multiplesimultaneous DOFs, especially where high-accuracy is desirable. It isperhaps not a coincidence that navigation systems have had their largestacceptance in cranial neurosurgery, where most applications involvespecifying a trajectory to a feature of interest without hittingcritical features. Often, the tip of the tool is pressed against theanatomy and pivoted, effectively decoupling the position and orientationplanning of the trajectory.

Autonomous robots have been applied commercially to joint replacementprocedures. These systems make precise bone resections, improvingimplant fit and placement relative to techniques that rely on manualinstruments. Registration is performed by having the robot touchfiducial markers screwed into the bones or a series of points on thebone surfaces. Cutting is performed autonomously with a high-speed burr,although the surgeon can monitor progress and interrupt it if necessary.Bones must be clamped in place during registration and cutting, and aremonitored for motion, which then requires re-registration. Deficienciesreported by users of these systems include the large size of the robot,poor ergonomics, the need for rigidly clamping the bone for the 45-60minutes required for registration and cutting, and the need forincreasing the incision by 50-100 mm to provide adequate access for therobot. Furthermore, autonomous robots generally function best in highlystructured environments, as evidenced by the rigid clamping of the bonesof interest and making larger incisions to keep soft tissue away fromthe robot.

Except for specific steps of some surgical procedures, modern surgeriesdo not tend to provide well-structured environments for autonomousrobots. A robot is generally not able to keep track of the surgicalstaff and instrumentation required to support a procedure. Althoughstrict management of the operating environment might make this possible,the complexity of the human body will always provide a high degree ofunstructuredness.

Robotic technology can also be used to improve upon standard practicewithout requiring autonomous operation. Notable commercial systems ofthis type include teleoperated robotic systems for laproscopic surgeriesranging from gall-bladder removal to closed-chest beating heart coronarysurgery. These systems provide a console for the surgeon that includes ahigh-fidelity display and a master input device. The slave robot iscoupled to the master and physically interacts with the anatomy. Thebenefits of these systems are primarily in providing an ergonomicworking environment for the surgeon while improving dexterity throughmotion scaling and tremor reduction. Although the master console wouldnormally be in the same room as the patient, an interesting byproduct ofthese systems is that they enable telesurgery. However, the robots haveminimal autonomy in these systems, which is not surprising given thecomplexity involved in manipulating and altering soft tissue.

SUMMARY OF THE INVENTION

It is often desirable to define objects with respect to images of ananatomy displayed using an image guided surgery system. For non-trivialobjects, or those with complicated two or three dimensional forms, itmay be difficult to present information in a manner that is simple for auser to understand. The local distance to a surface of interest, such asthe surface of the defined object, or to a desired position, the localpenetration distance of the surface of interest, or haptic repulsionforce, often provides the most useful information for augmenting theinteraction of the user with the image guided surgery system. The scalarvalue of the local distance may be conveyed to the user by visual,audio, tactile, haptic, or other means.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, the objectsand advantages thereof, reference is now made to the followingdescriptions taken in connection with the accompanying drawings inwhich:

FIG. 1 is a diagrammatic illustration of an exemplary operating room inwhich a haptic device is used with a computer-assisted surgery system;

FIG. 2 illustrates an exemplary haptic device being used in conjunctionwith a computer-assisted surgery system;

FIGS. 3A and 3B illustrate different types of haptic objects;

FIG. 3C is a flowchart of an exemplary method for intra-operative hapticplanning of a surgical procedure;

FIG. 4A illustrates the use of a dynamic haptic object for placement ofa haptic device;

FIG. 4B is a flowchart of a method for interactive haptic positioning ofa medical device coupled to a haptic device;

FIG. 5 illustrates the use of an exemplary haptic device in conjunctionwith a computer-assisted surgery system;

FIG. 6A illustrates an exemplary haptic device being used for hapticsculpting of physical objects;

FIG. 6B illustrates an exemplary haptic object for haptic sculpting ofphysical objects;

FIG. 6C is a flowchart of a method for dynamically modifying a hapticobject;

FIGS. 7A and 7B illustrate the use of an exemplary haptic device and asurgical tool to define a haptic object;

FIG. 8 illustrates the use of an exemplary haptic device as an inputdevice;

FIG. 9 is a flowchart of a representative method for using a hapticdevice as an input device;

FIG. 10 illustrates a system for conveying depth information during amedical procedure; and

FIG. 11 is a flowchart of a method for conveying depth informationduring a medical procedure.

DETAILED DESCRIPTION OF THE DRAWINGS

In the following description, like numerals refer to like elements.References to “surgeon” include any user of a computer-assisted surgicalsystem, a surgeon being typically a primary user. References to“surgical procedure” include any medical procedure, whetherinterventional or non-interventional, an interventional procedure beingtypically the primary procedure.

A haptic device is a mechanical or electro-mechanical device thatinteracts and communicates with a user, such as a surgeon, using sensoryinformation such as touch, force, velocity, position, and/or torque.Some robots may be used as haptic devices, though haptic devices mayinclude devices that are not necessarily considered to be robots in aconventional sense. Haptic devices typically have little autonomy.

In general, a component of interest may be optionally coupled to thehaptic devices. A component of interest may comprise a medical device,for example a surgical tool, a microscope, a laser range finder, acamera, a surgical light, an endoscope, an ultrasound probe, aradiotherapy device, interventional medical tools, rehabilitativesystems for physical therapy, and/or the like. The terms “medicaldevice”, “surgical device” and “surgical tool” are used interchangeablyherein.

For example, when used during surgery, such devices cooperatively hold asurgical instrument in conjunction with the surgeon. The surgeon movesthe surgical instrument with the assistance of, or input from, thehaptic device. Alternatively, in a teleoperation system, the hapticdevice may exclusively hold the surgical instrument. In such animplementation, the surgeon moves a “master” haptic device that iscoupled to a “slave” device in order to interactively manipulate thesurgical tool. In a teleoperation system, the master haptic device maybe physically separated from the surgical site to provide a moreergonomic or immersive working position for the surgeon and/or allow thesurgeon to perform the surgery remotely. In an impedance mode, a hapticdevice measures or senses the pose (position, orientation, velocity,and/or acceleration) of the surgical instrument and applies forcesand/or torques (“wrench”) to the instrument. In an “admittance” mode, ahaptic device measures the wrench at some location on the device (orsurgical instrument) and acts to modify the position of the instrument.There may be a static, quasi-static, or dynamic mapping between thesensed pose and output wrench. Common mappings may include wrenches thatresult from the tool interacting with “virtual” objects defined by orwith input from a user, which may include mathematical or simulatedmechanical constraints.

A “haptic object” is used herein to describe such a mapping. In somecases, a haptic object may only produce non-zero outputs for certainjoint angles of the haptic device, or only for certain endpointpositions and/or orientations of the haptic device. A haptic object maybe a smoothly time varying mapping and/or may only exist for certaintimes. A haptic object may have an associated spatial or geometricrepresentation that corresponds to locations where the mapping isdiscontinuous or has other properties that can be felt by the user wheninteracting with the haptic object. For example, if a haptic object onlyproduces non-zero outputs when the endpoint of the haptic device lieswithin a spherical region in space, then it may be useful to present acorresponding spherical representation to the user. However, a hapticobject may not necessarily have such a clearly defined boundary orsimilar internal structures. A haptic object may be active over theentire range of endpoint positions, endpoint orientations, and/or jointpositions of the haptic device or only a portion of these ranges. Theremay be multiple haptic objects active at any given time, possibly inoverlapping portions of space.

A “haptic cue” is used to describe an aspect of the mapping of a hapticobject. Having a cue may convey information or produce a desired effectwhen the user interacts with the haptic object. Haptic cues and hapticobjects do not necessarily correspond to user interface or softwareprogramming components in a particular embodiment and may be simply oneof many ways to design, implement, present to the user the mappingsbetween the inputs and outputs of the haptic device.

The reduction or elimination of autonomy increases the comfort level ofusers, such as surgeons. Any time a robot moves autonomously, thesurgeon is no longer in control and must simply observe the robot'sprogress. Robot motions have to be slow to provide adequate time for thesurgeon to respond should something unusual happen. If, however, a robotacts, at least mostly, in a passive manner, even if capable of activemotions, then the surgeon does not cede control to the robot.

Using a device capable of active motions in such a way that it only actslike a passive device from the user's perspective has advantages. Activeactuators can be used to counteract the effect of gravity, allowing agreater variety of mechanism designs. The device can be used in anautonomous mode for performing automated testing and service procedures.

FIG. 1 is a diagrammatic illustration of an exemplary operating room inwhich a haptic device 113 is used with a computer-assisted surgerysystem 11. Computer-assisted surgery system 11 comprises a displaydevice 30, an input device 34, and a processor based system 36, forexample a computer. Input device 34 may be any input device now known orlater developed, for example, a keyboard, a mouse, a trackball, and/orthe like. Display device 30 may be any display device now known or laterdeveloped for displaying two-dimensional and/or three-dimensionalimages, for example a monitor, a wearable display, a projection display,a head-mounted display, stereoscopic views, a display device capable ofdisplaying image(s) projected from an image projecting device, forexample a projector, and/or the like. If desired, display device 30 maybe a display device capable of displaying a holographic image. Ifdesired, display device 30 may be a touch screen and be used as an inputdevice.

Haptic device 113 is, in the illustrated example, a robotic device.Haptic device 113 may be controlled by a processor based system, forexample a computer 10. Computer 20 may also include power amplificationand input/output hardware. Haptic device 113 may communicate withcomputer-assisted surgery system 11 by any communication mechanism nowknown or later developed, whether wired or wireless.

Also shown in FIG. 1 is a storage medium 12 coupled to processor basedsystem 36. Storage medium 12 may accept a digital medium which storessoftware and/or other data. A surgical tool or instrument 112 is showncoupled to haptic device 113. Surgical tool 112 is preferablymechanically coupled to haptic device 113, such as by attaching orfastening it. However, if desired, surgical tool 112 may be coupled,either directly or indirectly, to haptic device 113 by any other method,for example magnetically. If desired, vacuum may be used to couplesurgical tool 112 to haptic device 113. Surgical tool 112 may behaptically controlled by a surgeon remotely or haptically controlled bya surgeon 116 present in proximity to surgical tool 112.

Haptic object 110 is a virtual object used to guide and/or constrain themovement and operations of surgical tool 112 to a target area inside apatient's anatomy 114, for example the patient's leg. In this example,haptic object 110 is used to aid the surgeon to target and approach theintended anatomical site of the patient. Haptic feedback forces are usedto slow and/or stop the surgical tool's movement if it is detected thata portion of surgical tool 112 will intrude or cross over predefinedboundaries of the haptic object. Furthermore, haptic feedback forces canalso be used to attract (or repulse) surgical tool 112 toward (or awayfrom) haptic object 110 and to (or away from) the target. If desired,surgeon 116 may be presented with a representation of the anatomy beingoperated on and/or a virtual representation of surgical tool 112 and/orhaptic object 110 on display 30.

When surgical tool 112 is haptically controlled by a surgeon remotely,for example when conducting a teleoperation, the surgeon controls themovement of the surgical tool using the master haptic device and/or areal or simulated display of the surgical tool, patient anatomy, and/oradditional haptic or visual objects designed to aid the surgicalprocedure. Haptic feedback forces may be transmitted by slave hapticdevice 113 to the surgeon at the remote location via the master hapticdevice to guide the surgeon. Alternatively, the haptic feedback forcesmay be generated at the master device and transmitted to the surgeondirectly. In some cases either the slave or master device may be apositioning device with little or no haptic capabilities.

The CAS system preferably includes a localization or tracking systemthat determines or tracks the position and/or orientation of varioustrackable objects, such as surgical instruments, tools, haptic devices,patients, and/or the like. The tracking system continuously determines,or tracks, the position of one or more trackable markers disposed on,incorporated into, or inherently a part of the trackable objects, withrespect to a three-dimensional coordinate frame of reference. Markerscan take several forms, including those that can be located usingoptical (or visual), magnetic or acoustical methods. Furthermore, atleast in the case of optical or visual systems, location of an object'sposition may be based on intrinsic features, landmarks, shape, color, orother visual appearances, that, in effect, function as recognizablemarkers.

Any type of tracking system may be used, including optical, magnetic,and/or acoustic systems, that may or may not rely on markers. Presentday tracking systems are typically optical, functioning primarily in theinfrared range. They usually include a stationary stereo camera pairthat is focused around the area of interest and sensitive to infraredradiation. Markers emit infrared radiation, either actively orpassively. An example of an active marker is a light emitting diodes(LEDs). An example of a passive marker is a reflective marker, such asball-shaped marker with a surface that reflects incident infraredradiation. Passive systems require an infrared radiation source toilluminate the area of focus. A magnetic system may have a stationaryfield generator that emits a magnetic field that is sensed by smallcoils integrated into the tracked tools.

With information from the tracking system on the location of thetrackable markers, CAS system 11 is programmed to be able to determinethe three-dimensional coordinates of an end point or tip of a tool and,optionally, its primary axis using predefined or known (e.g. fromcalibration) geometrical relationships between trackable markers on thetool and the end point and/or axis of the tool. A patient, or portionsof the patient's anatomy, can also be tracked by attachment of arrays oftrackable markers. In the illustrated example, the localizer is anoptical tracking system that comprises one or more cameras 14 thatpreferably track a probe 16. As shown in FIG. 1, cameras 14 may becoupled to processor based system 36. If desired, cameras 14 may becoupled to computer 10. Probe 16 may be a conventional probe now knownor later developed. If desired, the probe may be rigidly attached tohaptic device 113 or integrated into the design of haptic device 113.

If desired, in an implementation, processor based system 36 may comprisea portion of image guided surgery software to provide minimal userfunctionality e.g., retrieval of previously saved surgical information,preoperative surgical planning, determining the position of the tip andaxis of instruments, registering a patient and preoperative and/orintraoperative diagnostic image datasets to the coordinate system of thetracking system, etc. Image guided surgery using this method may not bepossible with the computer alone. As such, full user functionality maybe enabled by providing the proper digital medium to storage medium 12coupled to computer 36. The digital medium may comprise an applicationspecific software module. The digital medium may also comprisedescriptive information concerning the surgical tools and otheraccessories. The application specific software module may be used toassist a surgeon with planning and/or navigation during specific typesof procedures. For example, the software module may display predefinedpages or images corresponding to specific steps or stages of a surgicalprocedure. At a particular stage or part of a module, a surgeon may beautomatically prompted to perform certain tasks or to define or enterspecific data that will permit, for example, the module to determine anddisplay appropriate placement and alignment of instrumentation orimplants or provide feedback to the surgeon. Other pages may be set upto display diagnostic images for navigation and to provide certain datathat is calculated by the system for feedback to the surgeon. Instead ofor in addition to using visual means, the CAS system could alsocommunicate information in ways, including using audibly (e.g. usingvoice synthesis) and tactilely, such as by using a haptic interface ofdevice. For example, in addition to indicating visually a trajectory fora drill or saw on the screen, a CAS system may feedback to a surgeoninformation whether he is nearing some object or is on course with anaudible sound. To further reduce the burden on the surgeon, the modulemay automatically detect the stage of the procedure by recognizing theinstrument picked up by a surgeon and move immediately to the part ofthe program in which that tool is used.

The software module may be such that it can only be used a predefinednumber of times. If desired, the software module functions only whenused in conjunction with the portion of the image guided surgerysoftware that resides on computer 36. The software which resides oncomputer 36 in conjunction with the software on the digital mediumprocesses electronic medical diagnostic images, registers the acquiredimages to the patient's anatomy, and/or registers the acquired images toany other acquired imaging modalities, e.g., fluoroscopy to CT, MRI,etc. if desired, the image datasets may be time variant, i.e. imagedatasets taken at different times may be used. Media storing thesoftware module can be sold bundled with disposable instrumentsspecifically intended for the procedure. Thus, the software module neednot be distributed with the CAS system. Furthermore, the software modulecan be designed to work with specific tools and implants and distributedwith those tools and implants. Moreover, CAS system can be used in someprocedures without the diagnostic image datasets, with only the patientbeing registered. Thus, the CAS system need not support the use ofdiagnostic images in some applications—i.e. an imageless application.

An example of the illustrated robotic arm is a robotic arm manufacturedby Barrett Technology, and referred to as the “Whole-Arm Manipulator” or“WAM”. This robotic arm has a cable transmission, which provides highbandwidth, backdrivability, and force fidelity. However, other roboticdevices capable of impedance or admittance modes of haptic interactioncould be used. For example, direct-drive systems or systems with othertypes of low-friction transmissions or systems with a combination oftransmission types may also be well-suited to serve as a haptic devicefor surgical applications. Furthermore, the haptic device need notnecessarily take the form of a robotic arm. The WAM robotic arm has afour degrees of freedom of movement. However, it is augmented by a 1-DOFdirect-drive wrist for trajectory-based medical applications. Ifdesired, degrees of freedom may be added or removed without affectingthe scope of the illustrated invention.

Though it has some advantages, a cable transmission has somedisadvantages. It requires careful installation and maintenance toprevent the possibility of failure during a procedure. Furthermore, acable transmission is not as stiff as geared transmissions. Similardeficiencies may also be found in haptic devices using other types oftransmissions.

These deficiencies may be addressed by augmenting existing positionsensors that are mounted on drive motors with additional redundantsensors. These sensors may be of various types, including withoutlimitation rotary encoders or resolvers, tilt sensors, heading (compass)sensors, sensors that detect the direction of gravity, an optical,magnetic or acoustical tracking system (such as optical camera systemsof the type commonly used to track surgical instruments), or laser-basedposition sensing. The output of these sensors can be compared with theoriginal sensors to detect discrepancies that may indicate problems inthe transmissions or sensors. In addition, the added sensors can be usedto detect both low bandwidth deflections in the cable transmissions,which the system can then easily compensate for using well-known controltechniques. The sensor may also detect the high bandwidth deflections inthe cable transmissions, which can provide an additional input to theservo loop and permit improved stability of the servo system, usingwell-known control techniques for systems that include sensors on boththe drive and load sides of a transmission. The sensor can also improvethe accuracy of the determination of the pose of the arm by reducing oreliminating the effect of deflections of the arm links and/ortransmission. Such sensors could also be used to overcome similardeficiencies in robotic devices using other types of transmissionsystems.

When performing surgery, a haptic device capable of holding a tool, e.g.a drill guide or other similar constraint or attachment mechanism forsurgical tools is positioned relative to the patient such that it canattain the poses appropriate for a variety of approaches for aparticular procedure. It is also registered to the physical anatomy suchthat it can correlate information in diagnostic or planning imagedatasets, which can be two or three dimensional, to locations inphysical space using well-known registration techniques. The imagedatasets may be one or more images generated using for example, magneticresonance imaging (MRI), computer tomography (CT), positron emissiontomography (PET), magnetic resonance angiography (MRA), single photonemission computed tomography (SPECT), magnetic resonance venography(MRV), contrast enhanced MR venography (CEMRV), CT angiography, CTmyelography, MR angiography, MR myelography, fluoroscopy, opticalimaging, isotope imaging, ultrasound microscopy, laproscopic ultrasound,and MR spectrometry. Such images may include, for example, x-ray images,digital x-ray images, computer tomography images, MRI images, MRAimages, MR spectrometric images, PET images, MRV images, SPECT images,CEMRV images, CT angiographic images, CT myelographic images, MRmyelographic images, flair images, two-dimensional fluoroscopic images,three-dimensional fluoroscopic images, two-dimensional ultrasonicimages, three-dimensional ultrasonic images, ultrasound microscopyimages, laproscopic ultrasound images, optical images, isotopic images,laser depth maps, line arts, sketches, “cartoon” representations,holographic images, and/or the like.

Features to be avoided, such as blood vessels, tendons, nerves, andcritical areas of the brain can be automatically, semi-automatically, ormanually defined on the image datasets. Features targeted by theprocedure, such as tumors, osteophytes, anatomical targets fordeep-brain stimulation, biopsy sites, anatomical sites for implantplacement, or other regions of the anatomy can also be automatically,semi-automatically, or manually defined on the image datasets.

The image dataset(s), coupled with definitions of features to beavoided, can be used to create haptic “cues” that indicate to thesurgeon that a violation of sensitive anatomy is taking place. A generalfunction of these types of cues is to apply forces and/or torques thattend to repulse the haptic device from poses where an instrumentattached to the device would, for example, impact the defined criticalfeatures. Similarly, the image dataset(s), coupled with the definitionsof features to be targeted can also used to create haptic cues thatindicate to the surgeon that the desired target region would be reachedby the surgical instrument appropriately attached to the haptic arm. Ageneral function of these types of cues is to attract the haptic deviceto such poses or lock the haptic device into these poses once they areattained.

While the haptic device can be deployed as a fully integrated componentof a computer-aided surgery system, there are advantages to having thehaptic device act as an optional peripheral to such a system. The systemis then convenient to use for procedures that do not require the use ofthe haptic device. There are also development and architecturaladvantages to this approach. The haptic device will likely require areal-time operating system or special motion control hardware togenerate high-frequency updates for the haptic control system. Thecomputer-aided surgery system will have different requirements, such asfast graphics processing hardware and compatibility requirements with arange of user input and output devices, so that there are advantages ofhaving two computer systems to meet the differing uses. Separating thecomputer surgery and haptic arm components also has safety advantages.The haptic device therefore preferably contains only computing softwareand hardware that is necessary for ensuring high-performance, stable,and safe operation. The computer aided surgery system can containsoftware and hardware for connecting to a hospital network, displayingvarious graphical views, supporting various user input/output devices,managing libraries of implant and instrument databases, and/or any otherfunctionality useful in such a system. This architecture also allowsdevelopers with minimal knowledge of haptic systems to buildapplications that use the haptic device. The physical interface betweenthese two systems can be wired or wireless, such as a serial, USB, orother cable communications interface, or wireless ethernet, wirelessserial, infra-red or other wireless communications system. The softwareinterface between these systems would include a set of commands thatallows the computer aided surgery system to control operation of thehaptic device. For example, the computer-aided surgery system may send acommand to the haptic device requesting it to enter into a joystick-likeinput mode with certain stiffness parameters. The haptic arm systemchecks if the parameters are safe and otherwise acceptable, and thenenters into such a mode or responds with an appropriate error message.The computer-aided surgery system and haptic device may also beintegrated into a single system unit, or may be implemented using asingle or a multi-processor computing device. The CAS system, the hapticdevice and/or computer 10 may also be integrated into another piece ofequipment, such as an imaging equipment (e.g., fluoroscopy, CT, MR,ultrasound, and/or the like), an equipment cart in the room where themedical procedure is performed, and/or the like.

Referring to FIG. 2, representative “haptic object” 20 is atwo-dimensional virtual plane. However, it is only an example of hapticobjects generally, which may be zero (e.g. a point), one (e.g. a virtualline or path), two (e.g. a virtual plane or flat surface), or threedimensional (e.g. a virtual curved surface, a cube or other solidobject), and may have simple or complex geometric shapes. Haptic object20 is preferably defined with respect to the space of a physical object,such as patient anatomy 114. Haptic object 20 is defined to guide and/orconstrain the movement of haptic device 113. The distance between hapticdevice 113 and haptic object 20 is shown in FIG. 2 by X and the distancebetween patient's anatomy 114 and haptic object 20 is shown by X1.Haptic object 20 may be used in connection with generating forcefeedback on haptic device 113. The generation of force feedback may alsodepend on various factors, for example, the velocity at which hapticdevice 113 is approaching patient's anatomy 114, the position of hapticdevice 113, haptic object 20, and/or the like. An algorithm whichcomputes the current position of haptic device 113 relative to hapticobject 20 may be used to provide information to the surgeon about thelocation of haptic device 113 relative to haptic object 20. When hapticdevice 113 comes within a predefined distance of haptic object 20, astiffness parameter may be changed to make it more difficult to movehaptic device 113. If desired, force may be applied in a direction awayfrom anatomy 114 to resist the movement of haptic device 113 towardanatomy 114 or to move haptic device 113 away from anatomy 114.

It may not be appropriate to implement rigid haptic objects, such asvirtual surfaces and walls, in certain cases. A surgeon will lose theability to feel the anatomy in any direction that is rigidly constrainedby the haptic device. In many applications, precise localization ofanatomical features cannot be achieved by simply combining diagnosticdatasets with a tool tracking system or precision robotic devices.Changes in the anatomy after the diagnostic datasets are taken, unsensedmotion in the kinematic chain connecting the anatomical features ofinterest and the tracking system's camera or haptic device, registrationerrors, and inaccuracies in the localization devices will contribute topositioning errors. Although CAS systems may be used to position thesurgical tool very close to the target region, more accurate positioningis often difficult or prohibitively costly. In some medical procedures,such as pedicle screw placement in the upper thoracic and cervicalportions of the spine, deep brain neurosurgical procedures, etc., aslight inaccuracy may adversely affect the medical procedure beingperformed. Therefore, it is desirable in these types of procedures thata surgeon retain an ability to feel the anatomy.

Haptic devices can be used for registering patients to CAS systems anddiagnostic data sets of the patient's anatomy, for example, by attachinga probe and touching it to a few selected anatomical landmarks,implanted fiducials, or multiple points on a surface of interest. Theycan be used for haptic exploration of diagnostic datasets to augment thevisual display of this information. This exploration may occurintra-operatively while registered to the actual patient anatomy orpre-operatively in a purely virtual way. This haptic exploration isespecially useful for exploring complex three-dimensional structures,where the surgeon's highly developed sense of touch can be used toexplore complexities or subtleties of the dataset that may be difficultor impossible to display adequately on a two-dimensional or eventhree-dimensional visual display.

While performing traditional freehand surgery, surgeons rely on localanatomical features to ensure proper positioning of the surgical tool.If the ability of the surgeon to feel the patient anatomy is preserved,the surgeon can explore the local anatomy and correct these localizationerrors based on his expert knowledge of structures of interest. In thisway, the final positioning is determined by nearby anatomical featuresrather than a tracking system sitting across the operating room or arobot whose base may not be rigidly connected to the patient.

A portion of surgical tool 112 coupled with a haptic device, for examplethe tip of surgical tool 112, may be used to sense properties of thelocal anatomy. The properties of the local anatomy may be used toposition surgical tool 112 or to verify the proper positioning ofsurgical tool 112. The properties that may be sensed or monitored by thetool include electrical properties of the anatomy, force, pressure,stiffness, conductivity, etc. The information from the tip may beprovided back to CAS system 11. The information may then, if desired, becorrelated with information from diagnostic image datasets of thepatient. If desired, information from the tool may be used to augment orreplace the information from the image datasets. In either case theinformation may be used for better placement of surgical tool 112.

Location or position information of the tool may be sensed and providedback to CAS system 11 without the use of a separate sensor. The surgeonmay manually move surgical tool 112 to the desired position. Positioninformation of the tip of surgical tool 112 in the desired position maybe determined directly by CAS system 11 and/or computer 10 without theuse of a separate sensor. Other properties of the anatomy may be sensedby placing sensors at the tip of surgical tool 112. The output from thesensors may be provided back to CAS system 11 for processing.

The collected information may be used for a variety of purposes, such asalerting the user to registration errors, fully or partially correctingregistration errors, displaying graphical representations of theinformation on display device 30, defining haptic objects to assist theuser, displaying graphical representations of the information on displaydevice 30 superimposed over one or more images of the anatomy, and/orthe like. If desired, the collected information may be logged for use inmachine learning techniques.

The combination of a haptic device and a CAS system is also useful forcombining haptic exploration of diagnostic datasets and use of thehaptic device as a primary input device for planning. In this way,haptic exploration naturally leads the user to a suitable plan forperforming a procedure. Additionally, in some circumstances it ispossible to have the haptic device and the tool coupled with it in thecorrect position for performing a procedure as a result of thisexploration/planning process, eliminating the need to move the hapticdevice into position as a separate step.

Referring to FIG. 3A, it may be desirable in certain procedures toconfine the surgical instrument to a small working volume, in which caseit may stay within a working area inside a haptic object during theentire procedure. It may be necessary in certain cases to segment ordefine manually certain important features, but for most applicationsautomated segmentation of the diagnostic datasets will be sufficient forproviding appropriate haptic feedback.

In the illustrated embodiment, one or more attractive haptic objects areassociated with a target region for performing the surgical procedureand one or more repulsive haptic objects are associated with anatomicalfeatures to be avoided during the surgical procedure. For example, asshown in FIG. 3A, haptic object 22 defines a working area or volume forconstraining movement of surgical tool 112. On the other hand, as shownin FIG. 3B, haptic object 24 defines a working area or volume forconstraining movement of surgical tool 112 so that it is prevented fromcoming close to critical regions, such as nerves 25, organs 27, etc. Forexample, once the haptic objects are defined, the user performs surgicalplanning by pushing haptic device 113 around until a pose is found wherethe cues from the attractive haptic objects are active indicating thatsurgical tool 112, when attached to haptic device 113, would reach thetarget region, and where the cues from the repulsive haptic objects areinactive, indicating that surgical tool 112 would not penetrate any ofthe defined sensitive anatomical regions. In most cases, theserequirements will not fully constrain the pose of the arm and the usercan move the arm within this range of acceptable approaches based on anysecondary criteria a user finds appropriate. In some cases, the arm mayachieve an equilibrium state where multiple attractive or repulsivehaptic cues act in opposite directions. The user might mistake thisconfiguration to be an acceptable pose, even though the target regionmight not be reached or the critical anatomy regions might be violated.The user may be alerted to this situation in a number of ways, includingaudible or visual indicators, or by a haptic cue such as a vibration ofhaptic device 113. The user could then correct this situation by pushingthe haptic device away from this pose. Once in a pose satisfactory tothe user, haptic device 113 can be locked into position, using hardwarebrakes, control serving techniques, or any other appropriate method toprovide a stable physical reference for the surgical procedure.

If fine adjustments are desired, the haptic device can be operated usinga mode where motion scaling, constraints, or other methods are used tomake such corrections that might otherwise be beyond the dexterity ofthe surgeon. For example, a control servo can be enabled to lock thedevice to a certain finite stiffness at the approximate desired pose.The surgeon can then make fine adjustments to this pose using a varietyof methods. For example, the surgeon may use a touch screen, a keyboard,a mouse, a trackball or voice inputs. If desired, the surgeon may pushthe end of the haptic device in the desired direction. In response tothese inputs, the system would adjust the desired pose appropriately,possibly in small increments that would be difficult to achieve bydirect positioning of the haptic device. It may be desirable to lockonly a portion of the pose so that the surgeon can focus on a morelimited number of adjustments at one time. This fine adjustment mayoccur after the coarse haptic positioning is complete, simultaneous withthe coarse haptic positioning, or interleaved with the coarse hapticpositioning.

For example, selecting a trajectory for a cranial neurosurgicalprocedure such as a biopsy, tumor resection, or deep-brain stimulationis a complicated 3-D planning problem. The surgeon must find a path to atarget area while avoiding blood vessels and sensitive areas of thebrain. If these regions can be turned into repulsive haptic objects,planning such a procedure may be as simple as applying a hapticconstraint that keeps the trajectory of a tool guide passing through thetarget of interest, and allowing the user to pivot the device about thispoint until it settles into a suitable pose where none of the repulsivehaptic objects are violated.

FIG. 3C is a flowchart of an exemplary method 140 for intraoperativehaptic planning of a surgical procedure. Haptic device 113 is placed inthe operating room such that surgical tool 112 may be positioned over alarge portion of a clinically reasonable range of surgical approachesfor a given surgical procedure. Surgical planning using method 140 isperformed in the presence of the patient and preferably without surgicaltool 112 being coupled to haptic device 113. Surgical tool 112 may be anon-contact medical device, such as a diagnostic or therapeuticradiation source. If desired, surgical planning using method 140 may beperformed with surgical tool 112 coupled to haptic device 113 but beingin a retracted state. When surgical tool 112 comprises a non-contactmedical device, it is preferably in a disabled state. A representationof the anatomy of the patient to be operated on may be displayed ondisplay device 30 along with a “virtual tool”. The virtual tool may be ahigh-fidelity representation or a schematic representation of surgicaltool 112, such as an axis, a point, or other feature of surgical tool112. The virtual tool indicates relative to the anatomy of the patient,the position and/or angle of surgical tool 112 or some portion thereofif the surgical tool had been coupled to haptic device 113 in its normalor enabled state.

In step 142, haptic device 113 is registered to the anatomy of thepatient. If desired, the representation of the anatomy of the patientdisplayed on display device 30 may also be registered with the anatomyof the patient so that information in diagnostic or planning datasetsmay be correlated to locations in the physical space. Any method forregistering, now known or later developed, may be used. In step 144, thetarget region is defined. The target region may be, for example, atumor, an osteophyte, an anatomical target for deep-brain stimulation, abone channel, and/or the like. The target region may be defined in anymanner now known or later developed. For example, the user, such as thesurgeon, may manually identify the target region on display device 30.If desired, the surgeon may define the target region by touching one ormore points on the target region or circling the target region ondisplay device 30 with a tool. Alternatively, the surgeon may define thetarget region by pointing a tool mounting axis of haptic device 113 tothe target region or by using haptic device 113 as an input device.Preferably, the identified target region is automatically highlighted ondisplay device 30. The tool mounting axis of haptic device 113 may be ofany shape, for example curved, straight, and/or the like. Regardless ofthe manner in which the target region is defined, it is desirable thatonce defined, the target region be clearly displayed on display device30 for confirmation. One or more attractive haptic objects, such ashaptic object 22 of FIG. 3A, may be associated with the target region.

In step 146, anatomical obstacles to be avoided are defined. Theanatomical obstacles comprise features to be avoided during surgery,such as major blood vessels, tendons, nerves, critical areas of thebrain, organs, healthy bones or other tissues, and/or the like. Theanatomical obstacles may be defined in any manner now known or laterdeveloped. For example, the surgeon may manually identify the anatomicalobstacles on display device 30. If desired, the surgeon may define theanatomical obstacles by touching one or more points on the anatomicalobstacles or circling the anatomical obstacles on display device 30 witha tool. Alternatively, the surgeon may define the anatomical obstaclesby pointing the tool mounting axis of haptic device 113 to theanatomical obstacles or by using haptic device 113 as an input device.Preferably, the identified anatomical obstacles are highlighted ondisplay device 30. Regardless of the manner in which the anatomicalobstacles are defined, it is desirable that, once defined, theanatomical obstacles are clearly displayed on display device 30 forconfirmation. One or more repulsive haptic objects, such as hapticobject 24 of FIG. 3B, may be associated with the defined anatomicalobstacles. Preferably, each anatomical obstacle has one repulsive hapticobject associated with it, although if desired more than one repulsivehaptic object may be associated with an anatomical obstacle.

In step 148, haptic device 113 is positioned, preferably by the surgeon,such that if surgical tool 112 were coupled to haptic device 113 or ifsurgical tool 112 were in an operating state, then the appropriateportion of the surgical tool would have the desired relationship withthe target region. For example, when coupled to haptic device 113,surgical tool 112 would penetrate the target region. Surgical tool 112is in its operating state when it is coupled to haptic device 113 and isnot retracted and/or is not disabled. Step 148 is preferably performedwithout regard to whether or not the tool may intersect the anatomicalobstacles in this position. A virtual tool displayed on display device30 is such that it's position and orientation corresponds to theposition and orientation of surgical tool 112 if surgical tool 112 hadbeen mounted on haptic device 113 or if surgical tool 112 were in itsnormal operating state. Thus, the surgeon may position haptic device 113in the desired pose while viewing the display on device 30, such thatthe virtual tool has the appropriate relation with the target region.

In step 152, a determination is made as to whether the virtual tool isintersecting any anatomical obstacles. If the virtual tool is notintersecting any anatomical obstacles, then the process starting at step162 is executed. Otherwise, the process starting at step 154 isexecuted. In step 154, haptic cues are provided by haptic device 113 tothe user. The haptic cues may be provided to the user based on one ormore haptic objects, for example the attractive haptic object(s)associated with the target region and/or the repulsive haptic object(s)associated with the anatomical obstacles. The repulsive haptic object(s)generate forces and/or torques that guide haptic device 113 away fromposes where the virtual tool would intersect the anatomical obstacles.Preferably, the repulsive haptic cues are active when the virtual toolpenetrates the repulsive haptic objects or is in proximity to therepulsive haptic objects. The attractive haptic object(s) cause thehaptic device to generate forces and/or torques that guide haptic device113 toward poses where the virtual tool has the desired relationshipwith the target region.

It is possible that the position of haptic device 113 may be such thatcues from multiple haptic objects cancel each other out even though thevirtual tool may be violating the anatomical obstacles. As such, in step156, a determination is made as to whether haptic cues from multipleobstacles are canceling each other out. If haptic cues from multipleobstacles are not canceling each other out, then the process starting atstep 158 may be executed. If haptic cues from multiple obstacles arecanceling each other out, then in step 160, a special haptic cue, forexample a vibration, may be provided to alert the user of this situationand the process starting at step 158 may be executed.

In step 158, haptic device 113 is moved, preferably by the surgeon.Haptic device 113 is preferably moved based at least in part on thehaptic cues provided by haptic device 113 to the surgeon. The positionof surgical tool 112 had it been coupled to haptic device 113 is trackedby the virtual tool and displayed on display device 30. Preferably, theuser moves haptic device 113 until an equilibrium pose is found. In theequilibrium position, the cues created by the attractive haptic objectsare active and those created by the repulsive haptic objects areinactive. The process starting at step 152 may then be executed todetermine whether the virtual tool is intersecting any anatomicalobstacles.

In step 162, a determination is made as to whether the user is satisfiedwith the trajectory to the target region. The user may make thisdetermination by viewing the virtual tool relative to the target regionas illustrated on display device 30. If the user is not satisfied withthe position and/or the orientation of the virtual tool, then theprocess starting at step 158 may be executed. If the user is satisfiedwith the position and the orientation of the virtual tool relative tothe target region and the obstacles, then the process starting at step164 may be executed. The user may indicate its satisfaction in one ormore of a number of ways. For example, the user may issue a voicecommand to indicate that it is satisfied with the position andorientation of the virtual tool. If desired, the user may activate afoot pedal or a button associated with the computer-assisted surgerysystem or haptic device 113 to indicate its satisfaction. If desired,the user may indicate its satisfaction via a touch screen, a keyboard, amouse, and/or the like, associated with the computer-assisted surgerysystem or haptic device 113. In step 164, haptic device 113 may belocked in the current pose.

Once the pose of haptic device 113 is locked, the surgical procedure maybe performed, for example by coupling surgical tool 112 to haptic device113 or by placing surgical tool 112 in its fully functional oroperational configuration. Because the pose of surgical tool 112relative to the anatomy has already been determined with the aid of thevirtual tool, surgical tool 112 will achieve the desired position whenit is coupled to haptic device 113 or when it is configured for use.

The illustrated method for intra-operative haptic planning of a surgicalprocedure may be implemented in software, hardware, or a combination ofboth software and hardware. The steps discussed herein need not beperformed in the stated order. Several of the steps could be performedconcurrently with each other. Furthermore, if desired, one or more ofthe above described steps may be optional or may be combined withoutdeparting from the scope of the present invention. Furthermore, one ormore of the above described steps may be performed outside the operatingroom to save time spent in the operating room. For example, steps 144and 146 may be performed prior to bringing the patient into theoperating room and prior to step 142.

A technical advantage of this exemplary embodiment for intraoperativehaptic planning of a surgical procedure is that it provides for tightercoupling of the planning and execution phases of the surgical procedure.Planning for the surgical procedure is preferably performedintra-operatively with respect to the patient. Thus, when planning iscomplete, the haptic device is in position for executing the surgicalplan. No additional motion of the haptic device is required to initiatethe execution phase. Furthermore, by using a virtual tool to determinethe trajectory of the real surgical tool to the target region, injury toanatomical features may be avoided during the planning phase.

A haptic object may be of any shape or size. As shown in FIG. 4A, hapticobject 26 may be funnel shaped to guide a medical device, for example asurgical tool, coupled to haptic device 113 toward a target area onanatomy 114 of the patient. The path of the haptic object may depend ona surgical plan. An algorithm may be used to create the funnel shapedhaptic object illustrated in FIG. 4A. The information desired to createthe funnel shaped haptic object may be based on a surgical plan. Ifdesired, haptic object 26 may move with haptic device 113. This allowsguidance of the surgical tool toward the target area from the currentposition of haptic device 113. Thus, the surgical tool may be guidedtoward the target area from any position in proximity to anatomy 114.Furthermore, the surgical tool may be guided from a current pose to adesired pose.

Haptic object 26 may be of any shape, for example, a line, a curve, acylinder, a funnel, and/or the like. Haptic object 26 is, in theillustrated example, defined as a virtual pathway to facilitateinteractive positioning of haptic device 113 and/or surgical tool 112coupled to haptic device 113 at a desired position. Haptic object 26guides surgical tool 112 coupled to haptic device 113 from an initialposition and/or pose toward a target area and/or a desired pose relativeto anatomy 114 of the patient. If desired, haptic object 26 may guidesurgical tool 112 to the target area along a path or trajectory 28. Thepath or trajectory 28 from the initial position to the target area maydepend on the surgical plan. The path may be of any shape, for example astraight line, a curve, a funnel, a cylinder, and/or the like. Based atleast in part on haptic object 26, haptic forces are applied to hapticdevice 113 as the user moves the surgical tool or haptic device to guidethe user in moving the surgical tool 112 along path 28 toward the targetarea.

Haptic object 26 is preferably steerable or reconfigurable. For example,the haptic object may be defined to move or to change position and/ororientation as the haptic device (or the surgical tool or instrumentcoupled to it) moves. This allows, for example, the user to guidesurgical tool 112 toward the target area from almost any position inproximity to anatomy 114. This reconfigurability or steerability ofhaptic object 26 also allows the user to guide surgical tool 112 to thedesired pose from its current position and/or pose.

Haptic object 26 may also be allowed to move from a pre-defined path orposition in order to avoid obstacles, preferably without deviating fromthe target area. This is especially useful in avoiding obstacles in thepath of haptic device 113 that computer-assisted surgery system 11 maynot be aware of Thus, surgical tool 112 may be steered by the usertoward the target area without colliding with other surgical tools andequipment, the patient, or operating room staff.

Steering, moving or reconfiguring is, in a preferred embodiment, inresponse to application of a force or torque on the haptic device or thehaptic object that exceeds a threshold value. For example, if the userpushes haptic device 113 against the haptic object with a force thatexceeds a threshold, then the haptic object will be repositioned,reconfigured or modified to a new configuration based on the input forceor torque. Preferably, haptic object 26 moves in the direction of theforce or torque thereby providing an intuitive method for repositioningor realigning haptic object 26.

If desired, haptic object 26 may move to a new location if the targetarea is changed. Thus, as shown in FIG. 4A, haptic object 26 may bemoved from an initial position to a new position, as shown by hapticobject 26′, in response to a change in the target area.

In an alternative embodiment, haptic object 26 may be defined as virtuallinear or non-linear springs, dampers, clutches, and/or the like,logically applied to one or more joints of haptic device 113. One ormore joints of haptic device 113 may comprise virtual detentscorresponding to the final desired pose of haptic device 113.Preferably, standard joint-space control techniques are used toimplement the haptic objects at each joint and conventional inversekinematics techniques are used to determine the joint positionscorresponding to the desired Cartesian position/angle of the hapticdevice. The user may avoid obstacles by specifying the sequence in whichthe joints of haptic device 113 “lock” into their detents. The user maybe permitted to modify the selected sequence by “unlocking” jointsduring positioning of surgical tool 112, especially if the sequence isdetermined through a trial-and-error technique. Interactive unlocking ofa joint by the user may be based on the magnitude, duration or dynamicproperty of the force and/or the torque at that joint by the user. Agraphical user interface, a footswitch, a keyboard, a button, and/or thelike, communicatively coupled to haptic device 113 may be used to unlocka joint. If desired, once the desired pose is achieved, the ability tounlock the joints may be disabled to prevent inadvertent motion ofhaptic device 113.

In another alternative embodiment, haptic object 26 may be defined byvirtual linear or non-linear springs, dampers, clutches, and/or thelike, logically associated with one or more redundant degrees-of-freedomof haptic device 113. For example, if a haptic device comprising of fourjoints is used to position the tip of surgical tool 112, then the hapticdevice 113 may be moved along one of the degrees-of-freedom withoutaffecting the position of the tip. Haptic object 26 may be associatedwith the redundant degree-of-freedom to permit the user to interactivelymodify the position of haptic device 113.

FIG. 4B is a flowchart of a method 170 for interactive hapticpositioning of a medical device, for example surgical tool 112 mountedto haptic device 113, using a reconfigurable or steerable haptic object26, all as shown in FIG. 4A. If desired, the reconfigurability of thehaptic object may be user-configurable such that the user may turn thisfeature ON or OFF depending on the application or depending on the stepof a particular application. When the reconfiguration feature isenabled, method 170 is preferably executed periodically.

In step 172, a determination is made as to whether the medical device isin a desired pose. This determination may be made by using sensinginformation from one or more position sensors, such as encoders orresolvers, which may be integrated in the haptic device. If desired,this determination may be made by using sensing information from anexternal device, such as a laser interferometer, a camera, and/or othertracking device.

If in step 172, it is determined that the medical device is in thedesired pose, then in step 174, haptic interaction forces and/or torquesto maintain the pose of the medical device are determined. Thisdetermination may be made based at least in part on the position and/orvelocity of the haptic device and/or the medical device relative to thedesired pose. Any control algorithm now known or later developed may beused for this determination, for example, robust control, adaptivecontrol, hybrid position/force control, Proportional-Derivative (PD)control, Proportional-Integral-Derivative (PID) control, Cartesian basedcontrol, inverse Jacobian control, transpose Jacobian control, and/orthe like. The determined haptic interaction forces and/or torques may betransformed and provided to the haptic device. If in step 172, it isdetermined that the medical device is not in the desired pose, then instep 176, haptic interaction forces and/or torques to maintain themedical device within a haptic object are determined so that the medicaldevice may be guided toward the target area. In step 178, adetermination is made as to whether the result of at least one scalarvalued function of the haptic interaction forces and/or torquescalculated in step 176 exceeds at least one reconfiguration threshold.The reconfiguration threshold may be user-configurable. A scalar valuedfunction computes a value based on one or more input values. In anexemplary embodiment, the scalar valued function may be the square rootof the sum of the squares of the input values. A scalar valued functionmay be applied to one or more haptic interaction forces to provide ascalar value. The resulting scalar value may be compared to thereconfiguration threshold. Dynamic properties of the haptic interactionforces and/or torques, such as direction, duration, and/or the like, mayalso be considered.

If the result of none of the scalar valued functions exceeds thereconfiguration threshold, then the process ends. Otherwise in step 180,haptic object 26 is modified based at least in part on the hapticinteraction forces and/or torques. For example, if the surgeon guidesthe haptic device such that the haptic device in effect pushes againstthe haptic object, the value of the scalar valued function of the hapticinteraction forces and/or torques generated to keep the haptic devicewithin the haptic object may exceed the reconfiguration threshold. Insuch a case, it is desirable that the haptic object be modified, forexample in the direction of the force applied by the surgeon such thatthe surgical tool is maintained within the haptic object. Themodification of the haptic object may comprise changing the size of thehaptic object, changing the shape of the haptic object, pivoting thehaptic object along the target area of the patient's anatomy, and/or thelike.

A technical advantage of this exemplary embodiment for interactivehaptic positioning of a medical device is that by modifying a hapticobject based on the haptic interaction forces and/or torques, greaterflexibility is provided to the surgeon. Thus, the surgeon may approachthe target area without colliding with other surgical tools andequipment, the patient or operating room staff, and still be providedwith haptic cues to enable the surgeon to guide the surgical tool to thetarget area.

The illustrated method for interactive positioning of a haptic deviceusing a reconfigurable (repositionable, steerable) haptic object may beused in any situation where it is desirable to move the haptic device,optionally coupling a component of interest, such as a medical device,for example a surgical tool, and/or the like, within a cluttered orsafety-critical environment. If desired, the haptic device itself may bethe component of interest. The illustrated method may be used in avariety of applications, such as a procedure where virtual constraintsand/or haptic cues are used to move the component of interest into apredefined location and/or orientation and safety or other concerns makeautonomous device motions undesirable. For example, the method may beused in an implant placement procedure, a biopsy procedure, depositionof therapeutic implants, diagnostic palpation of internal or externalanatomy, tumor removal, radiation therapy, artistic or commercialsculpting, artistic or commercial painting, scientific or engineeringexperiments, such as surface digitizing, sample collection, circuitboard probing, manual assembly, fabrication or testing of mechanicaland/or electronic components or assemblies, material handling, and/orthe like.

For rehabilitation and/or physical therapy applications, a haptic devicemay be coupled to the patient using an orthotic device, which mayrequire the patient to grasp a handle. In such an embodiment, the hapticdevice may be coupled to a computer system having a user console. Thecomputer system may or may not be a CAS system, but may be a computersystem designed for rehabilitative or physical therapy applications. Ifdesired, the computer system may be integrated with computer 10. Theorthotic device may have straps, braces, shells, or cast features toprovide a firm or loose connection as desired. The orthotic deviceallows the haptic device to guide, monitor, and/or assist rehabilitativemotions or other exercises. For example, the patient or a therapist maycouple the patient's arm or leg to the haptic device and lead it througha desired motion while the haptic device records the properties of themotion. The motion can then be repeated multiple times without theassistance of the therapist. The haptic device may also be used tomonitor the patient's efforts to move by noticing how much effort isrequired to move the patient, or through the use of force sensingdevices which may be coupled to the haptic device at or near thelocation where the patient interfaces with the haptic device. The hapticdevice may also be used to simply constrain the patient's motion to thedefined path which requires the patient to advance along the definedpath using their own strength. Modes where there is a shared effortbetween the patient and the haptic device may also be advantageous. Itis desirable that when used in this manner, the haptic device operate ina safe manner because it is so close to the patient, who may have onlypartial function in one or more extremities. It may be undesirable forthe haptic device to move to new positions automatically orautonomously. However, it may be desirable to reposition the hapticdevice, for example to permit initial attachment to or grasping by thepatient, so that the haptic device may be moved to different startingpositions between different exercises or repetitions of the sameexercise, or in the course of performing the rehabilitative motions orexercises. A physical therapist may provide the interactive input forrepositioning the haptic device. If desired, the patient may providesuch input while interfacing with the haptic device.

The illustrated method for interactive haptic positioning of a surgicaltool using a reconfigurable or steerable haptic object may beimplemented in software, hardware, or a combination of both software andhardware. The steps discussed herein need not be performed in the statedorder. Several of the steps could be performed concurrently with eachother. Furthermore, if desired, one or more of the above described stepsmay be optional or may be combined without departing from the scope ofthe present invention.

Referring now to FIG. 5, when the user interacts with a haptic object,such as haptic object 20, it is sometimes desirable to know themagnitude of forces applied to the haptic object or the amount that areal or virtual tool or implant is penetrating the haptic object. Fornon-trivial haptic objects, or those with complicated two or threedimensional forms, it may be difficult to present this information in amanner that is simple for the user to understand. However, the desirablepiece of information is often the local penetration distance or hapticrepulsion force. While these can be up to three-dimensional vectorquantities, the magnitude (or length) of such vectors, possibly in thedirection of a local unit normal of the haptic object, are most usefulfor augmenting the haptic interaction of the user. These magnitudes aresimple one-dimensional quantities and can be conveyed to the user in avariety of methods, including meters, dials, numerical displays, graphs,and other visual methods, but also with audio, tactile, haptic, or othermeans.

Though a complete message is conveyed directly by haptic device 113 tothe hand of the surgeon, a visual or audible display can be used tosupport rich interactions between the user and the system. For example,well known and commercially available speech recognition techniques canbe used to provide a verbal method for the user to communicateinformation or instructions to the computer aided surgery system. Speechoutput from the computer aided surgery system 11 can also be used forcommunicating information to the user including status information,warning messages, event notification, and responses to user queries,whether communicated verbally or through some other method. Computermonitors, projection displays, wearable displays, head-mounted displays,stereoscopic views, holographic displays, and/or other visual displaydevices can be used to provide schematic anatomic representations,images of diagnostic datasets, instructions or guides for the surgicalprocedure, depictions of virtual and haptic objects, system statusinformation, patient information, and other information that is easilycommunicated over a visual display. Any other input or output devicecould similarly be used to augment the haptic interaction between theuser and the computer surgery system.

A visual and/or audio display of the penetration into a haptic object ofa predetermined stiffness of a surgical device's depth, force and/orvelocity is provided. The haptic object is based upon information fromthe computer-assisted surgical system. The display is one-dimensional inorder to facilitate the communication of the local penetration magnitudeof the surgical device into the haptic object.

During surgery, the haptic device may be used to enhance the performanceof the surgeon in, for example, such tasks as holding a tool steady,making straight cuts, or moving a tool tip along a path or surface. Thehaptic device can replace mechanical cutting jigs and alignmentapparatus used for aiding in the placement of and preparation of anatomyfor implanted medical devices. Virtual haptic surfaces may be used toreplace physical cutting blocks. The virtual haptic surfaces in thisinstance are preferably software entities that can be easily and cheaplycreated from the models of the implant. The virtual haptic surfaces canbe created with curved shapes, which more closely match the underlyinganatomy and enable implant designs that require less bone or tissueremoval.

Sculpting of a physical object, such as a bone, frequently requiresmultiple planar features to be created in the bone and/or on the surfaceof the bone. A haptic object may be defined to assist in such sculpting.The shape of the defined haptic object may correspond substantially tothe desired resulting shape of the physical object after sculpting. Thephysical object and the haptic object may have segments or surfaces withabrupt transitions and/or may have portions with short radius ofcurvature. As such, it is possible that a surgical tool coupled to thehaptic device and being used to sculpt the physical object may abruptlyfall off one segment causing unintentional damage to the physical objector other objects in the vicinity of the physical object, or bedistracting or disturbing to the user. A segment may be one-dimensional,two-dimensional or three-dimensional.

In order to address this problem, haptic object is dynamically modifiedduring sculpting in order to prevent the surgical tool or the hapticdevice from following an abrupt transition from one segment of thehaptic object to another segment. Preferably, the haptic object remainsin the modified form only so long as it is desirable to prevent abrupttransitioning of the surgical tool or the haptic device from one segmentto another. Once the cutting or portion thereof is complete, the hapticobject may be returned to its original configuration, for example to itsoriginal shape, size, orientation, and/or the like. The modification ofthe haptic object may comprise creating another haptic segment thatprevents the surgical tool from following an abrupt transition from onesegment of the haptic object to another segment of the haptic object,modifying an existing segment of the haptic object, for example byextending the existing segment beyond its boundary, and/or the like.

FIG. 6A illustrates an exemplary haptic device being used for hapticsculpting of physical objects with high curvature. FIG. 6B illustratesan exemplary haptic object 20 for haptic sculpting of physical objectswith high curvature. Referring now to FIGS. 6A and 6B, in certain cases,the desired shape for an anatomical region to be prepared with the aidof a haptic object may include sharp external edges. It is difficult toproperly execute cuts without slipping off these edges, resulting inrounding of the edges and other unwanted artifacts in the resultingcontour of the anatomy. An improved method for preparing these types ofshapes involves dynamically enabling and disabling portions of thehaptic surface. In particular, this method is helpful if a haptic objectcontains at least one sharp external edge where the local angle betweenthe two portions joined by the edge as depicted in FIGS. 6A and 6B isless than 180 degrees. The method includes a way of selecting one ofthese portions, which may include any of the user input modalitiesmentioned herein, but the preferred method is based on proximity to thehaptic object. When one of the portions is selected, that portion of thehaptic object is extended beyond the joining edge to provide acontinuous guide surface. When the extension is no longer required, theuser can return the haptic object to its original configuration bymoving the haptic arm away from the portion or using any other inputmodality.

For example, in a total or unicompartmental knee replacement procedure,multiple planar cuts are often required to prepare the femur for thefemoral implant. A haptic object is defined in software that containsportions closely related to the desired femoral cuts. In experiments,when the user attempts to resect the bone using a cutting burr mountedin the haptic arm using the full haptic object, it is difficult to makethe straight cuts without slipping from one portion to another andfrequently moving the burr beyond the desired region. This slipping mayresult in damage to tendons, blood vessels, ligaments, and otherstructures and distract the user. If instead, each cutting plane of thehaptic object is extended when the user brings the cutting burr withinclose proximity to that portion, it is much easier to create straightcuts without moving beyond the local anatomical site. The portion isreturned to its original extent by simply moving back away from it, atwhich time the user can bring the cutting burr into contact with any ofthe other portions to extend them in a similar manner. While footpedals, voice commands, or other input modalities can be used to controlthe extension of each plane, controlling them in the preferred mannerdescribed previously requires no additional hardware and is extremelysimple for the user. However, a visual display of the haptic object andthe extended portion is also helpful for helping the user to understandmore complex haptic objects, especially where their view of the cuttingis limited due to obstructions or a minimally-invasive technique.

FIG. 6A shows an exemplary system for dynamically extending a hapticobject. A representation of the physical object, for example the anatomyof the patient to be sculpted, may be displayed on display device 30.The representation of the physical object may comprise a two-dimensionalor three-dimensional drawing or image. The image could be, for example,a two-dimensional medical diagnostic dataset or a three-dimensionalmedical diagnostic dataset of the patient. In FIG. 6A, haptic object 20includes two different portions (20′ and 20″) divided by a well definededge 21. When haptic device 113, the surgical tool, or the virtualsurgical tool comes within a predefined distance, say R1, of oneportion, say portion 20′, that portion of haptic object 20 is activated.If desired, the activated portion of the haptic object 20 may beextended as shown by the broken lines 23 in FIG. 6A. When haptic device113 moves to within a predefined distance of another portion, sayportion 20″, the new portion of haptic object 20 may be activated. Ifdesired, the newly activated portion of haptic object 20 may beextended.

It is desirable that haptic object 20 with high curvature be logicallydivided into or be approximated by a plurality of portions or segmentswithout high curvature. For example, as shown in FIG. 6A, haptic object20 may be logically divided into a plurality of portions 20′ and 20″separated by an edge 21. Although, it is preferable to logically dividea haptic object into a plurality of segments, the haptic object itselfmay be defined using a logical combination of a plurality of segments.For example, a plurality of segments may be initially defined and thehaptic object may be defined as a logical combination of one or more ofthe plurality of segments. If desired, the haptic object may comprise aregular or irregular arrangement of volume elements, or voxels, some orall of which may be labeled. It may be desirable to only label thevoxels on the surface of the object in this manner.

FIG. 6C is a flowchart of a method 120 for dynamically modifying ahaptic object, such as haptic object 20 of FIGS. 6A and 6B. If desired,the dynamic modification feature may be user-configurable such that theuser may turn this feature ON or OFF depending on the application ordepending on the step of a particular application. When the dynamicmodification feature is enabled, method 120 is preferably executedperiodically.

In step 122, a determination is made as to whether a configuration ofthe haptic object, say haptic object 20, has already been modified, forexample by modifying a segment of the haptic object or by adding a newsegment. In the preferred embodiment, the value of a configuration flagmay be checked to determine if haptic object 20 has already beenmodified. If haptic object 20 has not already been modified, then instep 124, a determination is made as to whether one or more criteria formodifying the configuration of haptic object 20 is satisfied. Thecriteria may be proximity of surgical tool 112 coupled to haptic device113 to haptic object 20, penetration of haptic object 20 by surgicaltool 112, gestural motions of surgical tool 112, gestural or othermotion of surgical tool 112 relative to the position of haptic object20, a fixed or variable time period, detection of an unwanted slippageover edge 21, and/or the like. If desired, the criteria may be proximityof the representation of surgical tool 112 to haptic object 20,penetration of the boundaries of haptic object 20 by the representationof surgical tool 112, gestural or other motion of the representation ofsurgical tool 112 relative to the position of haptic object 20, and/orthe like. When modification of the configuration of haptic object 20comprises modifying a segment of haptic object 20, preferably the samecriteria is used to determine if any of the segments should be modified.However, if desired, different segments may be modified based ondifferent criteria. In such an embodiment, each of the plurality ofsegments may have one or more criteria associated with it.

If in step 124, it is determined that at least one criteria formodifying the configuration of haptic object 20 is satisfied, then instep 126, the segment to be modified is selected. Alternatively, asegment in proximity to which a new haptic segment is to be created maybe selected in step 126. In an alternative embodiment, the processstarting at step 126 may be executed if a predefined logical combinationof a set of criteria are satisfied. Preferably, the segment that isclosest to haptic device 113 is selected. However, if desired, othercriteria may be used to select a segment. For example, if surgical tool112 has crossed an edge between two or more segments since the last timemethod 120 was executed, then one of the segments associated with theedge that was crossed may be selected. Alternatively, the segment beingpenetrated by surgical tool 112 may be selected. In step 128, theconfiguration of the selected segment is modified, preferably byextending the selected segment in a desired direction of movement ofhaptic device 113. The configuration flag may be set to indicate thathaptic object 20 has been modified.

The method for modifying the configuration of the selected segment ispreferably based at least in part on the manner in which the hapticobject is represented. This representation may be based on surfacepolygons, voxels, non-uniform rational B-splines (NURBs), constructivesolid geometry, and/or any other method for representing haptic objectsnow known or later developed. The modified segment may be represented inany manner which may or may not be the same as those used to representthe original haptic object. Preferably, the selected segment is extendedsuch that the extended portion is continuous with the segment along oneof its high curvature edges. The extension may be flat or curved. Thesegment may be extended a fixed or variable distance beyond the originalsegment, or could be extended to intersect another portion of the hapticobject or the edge of a workspace. The method used for extending thesegment depends on the method used for representing the extension. Forexample, if a haptic object is represented with surface polygons, thenthe polygons that lie within the segment of interest and adjacent to oneof its boundaries are identified. A neighboring segment that lies beyondthe original segment and has the same normal direction as the originalpolygon may be enabled. For a voxel representation, the voxels may belabeled to indicate whether they behave as solid or filled regions ofspace for configurations of the haptic object with different extendedsegments, which may be automatically, semi-automatically, or manuallydesigned. The selected neighboring segment may be added to the hapticobject. Thus, as illustrated in FIG. 6A, if portion 20′ of haptic object20 is the selected segment, then portion 20′ may be extended beyond itsoriginal boundary, for example as shown by broken lines 23.Alternatively, if desired, a new haptic segment may be created inproximity to the selected segment.

In step 130, haptic interaction forces and/or torques for the hapticobject are calculated. The haptic interaction forces and/or torques maybe transformed and provided to haptic device 113. For example, it may bedesirable to compute appropriate forces and torques for the actuators ofthe haptic device to apply such that the desired haptic interactionforces and/or torques will be produced. In some cases, it may bedesirable to alter position or velocity commands to the actuators toproduce the desired effect. The haptic interaction forces and/or torquesfrom the selected segment may be used to guide haptic device 113 in adesired direction away from, toward, or aligned with physical object 114to be sculpted. The haptic interaction forces and/or torques may berepulsive, attractive, frictional, viscous, impulsive, detent,regulatory (for example designed to maintain cutting speeds or feedrates), and/or the like. If desired, the haptic interaction forcesand/or torques may be calculated using a mathematical, control theory,or machine learning algorithm.

If in step 124, it is determined that the criteria for modifying theconfiguration of haptic object 20 is not satisfied, then the processstarting at step 130 may be executed.

If in step 122, it is determined that the configuration of haptic object20 has already been modified, then in step 134, a determination is madeas to whether one or more predefined criteria for maintaining hapticobject 20 in the modified configuration is satisfied. These criteria mayor may not be the same as those considered when the configuration ofhaptic object 20 was initially modified. Preferably, if at least onecriterion for maintaining the haptic object in the modifiedconfiguration is satisfied, then the process starting at step 130 may beexecuted. Otherwise, in step 136, the modified haptic object is returnedto its original configuration. The configuration flag may be reset toindicate that haptic object 20 has not been modified. After execution ofstep 136, the process starting at step 130 may be executed. In analternative embodiment, the process starting at step 130 may be executedif in step 134 it is determined that a predefined logical combination ofa set of criteria are satisfied.

As illustrated in FIG. 6A, when haptic device 113 or surgical tool 112coupled to haptic device 113 comes within a predefined distance R1 ofone portion of haptic object 20, say portion 20′, that portion of hapticobject 20 may be activated and modified such that it extends beyond itsoriginal boundary as shown by dashed lines 23. While haptic device 113or surgical tool 112 is in close proximity to portion 20′ or maintainscontact with portion 20′, portion 20′ remains modified. Surgical tool112 may be used during that time to sculpt the portion of physicalobject 114 corresponding to portion 20′ to a desired shape. When thesculpting of the portion of physical object 114 corresponding to portion20′ is completed, the user may move haptic device 113 away from portion20′. Portion 20′ may then be returned to its original configuration.When haptic device 113 or surgical tool 112 moves to within a predefineddistance of another portion of haptic object 20, say portion 20″,portion 20″ of haptic object 20 may be activated and modified such thatit extends beyond its original boundary.

The illustrated method for dynamically modifying a haptic object may beused in a variety of applications, such as any procedure where a virtualconstraint and/or haptic cues are used to guide a user using a hapticdevice for sculpting a physical object or shape that has high curvature.For example, the method may be used in fabrication of components forconsumer or industrial products, for the reproduction or creation ofartistic pieces, such as sculptures, for shaping bones in an orthopedicprocedure, and/or the like.

The illustrated method for dynamically modifying a haptic object may beimplemented in software, hardware, or a combination of both software andhardware. The steps discussed herein need not be performed in the statedorder. Several of the steps could be performed concurrently with eachother. Furthermore, if desired, one or more of the above described stepsmay be optional or may be combined without departing from the scope ofthe present invention.

A technical advantage of this exemplary embodiment for dynamicallymodifying a haptic object is that the sculpting of the physical objectmay be performed in a more controlled manner. Thus, during a surgicalprocedure, unintentional damage to parts of the body may be avoided andthe user can feel more comfortable using the system. Another technicaladvantage is that the user does not have to move its attention away fromthe working volume when switching from one segment to another segment ofthe haptic object. Yet another technical advantage is that shapes withhigh curvature may be operated on more easily than if only the entirehaptic object were used.

FIG. 8 illustrates the use of an exemplary haptic device 113 as an inputdevice. Haptic device 113 and a haptic object 20 in real space areillustrated. Haptic device 113 may also be used as an input device,allowing information to pass from the user to CAS system 11, andproviding functionality similar to common user interface devices, suchas a mouse, touchpad, keyboard, joystick, flight controller, hapticjoystick, or any other input device. When used as an input device, itmay be used for defining anatomical reference geometry, manipulating theposition and/or orientation of virtual implants, manipulating theposition and/or orientation of surgical approach trajectories,manipulating the position and/or orientation of bone resections, and theselection or placement of any other anatomical or surgical feature.Haptic device 113 may also be used for more generic user interfacefunctions, including but not limited to, moving a cursor 31 (FIG. 8),selecting buttons or other similar user interface objects, selectingpull-down menus, manipulating on-screen dials, knobs, and othercontrols. When in this user-input mode the haptic device can beconstrained to move in only certain directions which may be definedrelative to the position of a predetermined portion of the hapticdevice, relative to the position of the patient or a portion of thepatient anatomy, or relative to images or 3-D models of schematic,virtual, atlas, or actual patient anatomical features. The predeterminedportion of the haptic device may be capable of moving.

As illustrated in display 30 of FIG. 8, haptic device 113 may be used asan input device to change the position, shape, size, etc. of hapticobject 20. An example of an application of haptic device 113 used inthis mode is planning the placement of a knee implant. After acquiringappropriate anatomical images of the anatomy of interest, the computersurgery system enters a mode where a cursor appears on a display visibleto the user. The user grasps the arm to move the position of the cursor,possibly in multiple views. When satisfied with the position of thecursor, the user fixes it in the desired position through the use of afoot pedal, button, wired or wireless control pendant, voice command, orother input, or through the application of a force or torque to thehaptic arm, or moving the haptic arm in a distinctive gesture, such as atap, twist, or other gesture that is easily distinguishable from theuser interactions during the cursor positioning. After the firstposition is set, a second cursor is used to define the endpoint of aline connecting to the two or three-dimensional position of the firstcursor. The second cursor is moved, as above, to define an anatomicalaxis of the femur bone and its position is fixed using one of the abovementioned techniques. The two or three dimensional position andorientation of the implant can then be manipulated by the user using thehaptic device as an input device. The implant is constrained by thesystem such that one of its surfaces is perpendicular to the anatomicalreference line, but its position and orientation can be adjusted by theuser. It is also possible to allow deviations from the anatomical axis,possibly coupled with displays of such deviations relative to anatomicalreference frames familiar to the user. For example, the varus/valgusangle of the implant relative to the anatomical reference line can beadjusted and displayed to allow appropriate alignment of the kneeimplants. This general technique can be adapted to plan the approachand/or placement of minimally invasive hip and knee implants, traumafixation pins, pedicle screws, biopsy needles, radioactive beads,radiotherapy beam emitter, or any other medical device.

With a haptic device, the surgeon can use tools identical or verysimilar to those used in standard practice. By exploiting the hapticfeatures of the device, the need for awkward teach pendants or GUI-basedrobot controls may be reduced or eliminated. Switching between freehandand assisted steps of a procedure is quickly performed by simply pushingthe device out of the way, similar to familiar operating room objectssuch as microscopes and overhead lights. While the systems may beinternally complex, the surgeon must be shielded from this complexity sothat he can focus all of his attention on his patient.

For example, the haptic arm can hold itself at a reference positionusing a joint-based or Cartesian control algorithm. The user appliesforces and/or torques to the arm, either on an interaction handle orend-effector or at any point on the arm, which cause the arm to deflectfrom the reference position. The amount and direction of the deflectionis continuously communicated to the computer system to modify theposition of any desired virtual reference geometric feature or userinterface object.

In another example, the haptic arm can hold itself at a referenceposition using a joint-based or Cartesian control algorithm but with twodegrees of freedom left unconstrained. The user can then move the arm inthe unconstrained directions to provide two-dimensional control of auser-interface object, such as a cursor, implant, or other geometric orvirtual surface entity. A similar technique can be used for one degreeof freedom manipulation of objects, such as user interface slider bars,implant lengths, positions of objects along a reference trajectory, orany other one-dimensional control such as audio volume, imagebrightness, object scaling, image zooming, and the like. A similartechnique can be used for higher than three degree of freedompositioning of implants or virtual or haptic objects. The hapticobject's position may also be constrained relative to any relevantanatomical features for a particular application. For example, a kneeimplant may be constrained to have the proper alignment relative to theanatomical axis of the leg, or to achieve proper ligament balance, butwith the other degrees of freedom controllable by the user in the mannerdescribed above.

The stiffness or damping of the control algorithm may vary in differentdirections to indicate preferential directions of motion which may bealigned with any direction as described in the previous paragraph. Thisstiffness variation may include zero stiffness along certain directionsor may lock the user to the preferred directions once the deviation fromthe reference position exceeds some threshold value. This stiffnessvariation assists with simplifying the planning process by allowing theuser to focus their attention on a limited number of degrees of freedomat a time. For example, the user may set the position of an implantalong one or two directions first, then set the position of the implantalong an additional direction or directions without disturbing the setdirections.

The stiffness and damping variations can occur automatically dependingon the physical interaction of the user with the haptic device and doesnot require the use of another input device such as a voice command,control pendant, or foot pedal. Any such simplification has benefits inreducing service costs, simplified system use, and improved safety. Thisgeneral method of planning also allows the surgeon to perform planningwithout having to leave the normal operating position to interact withthe computer-aided surgery system or requiring an assistant to controlthe computer-aided surgery system or requiring the introduction ofadditional input devices other than the haptic device which is alreadybeing used for execution of the surgical plan. An additional benefit ofthis use of a haptic device is that the motion of the controlled objectcan be scaled relative to the motion of the arm, so that it can bepositioned to a precision better than the user can position a realobject, eliminating the deleterious effects of the user's hand tremorand any force disturbances arising from friction, backlash, magneticdetent forces, and other force disturbances arising from the haptic arm.It should be noted that the primary function of the object controlled bythe haptic device is something other than monitoring the pose of thehaptic device or monitoring the pose of a component of interest that mayor may not be coupled to the haptic device.

FIGS. 7A and 7B illustrate the use of a haptic device and a surgicaltool to define a haptic object. In the illustrated example, hapticdevice 113 is being used as an input device to define haptic object 182.In order to use haptic device 113 as an input device to define hapticobject 182, the user grasps surgical tool 112 coupled to haptic device113. If desired, the user may grasp haptic device 113 itself. Usingsurgical tool 112 the user traces the boundaries of a desired region,for example a portion of the anatomy with respect to which the hapticobject is to be defined. The user may trace the boundary, for example bytouching the end of surgical tool 112 to portions of the desired regionof the anatomy. The motion of surgical tool 112 may be recorded and thelocations of the endpoints traced by the user computed. The geometryand/or location of haptic object 182 may be determined based at least inpart on the location of the endpoints. A haptic device creation mode maybe used to specify the desired shape of the haptic object. For example,to create a cylindrical haptic object that corresponds to a resectedportion 184 of anatomy 114, the user can trace a plurality of points onthe boundary of resected portion 184. An appropriate cylindrical hapticobject may be created using any technique now known or later developed.

Material and other properties of the anatomy may be defined by probingthe anatomy. For example, surgical tool 112 may include a forcemeasurement device coupled to the tip of surgical tool 112.Alternatively, if desired, instead of surgical tool 112, a probecomprising a force measurement device may be coupled to haptic device113. When the user interfaces the force measurement device against aportion of anatomy 114, the force may be measured by the forcemeasurement device. The measured force may be displayed as a function ofthe distance the anatomy moves, if any, upon application of the force.The stiffness of that portion of anatomy 114 may be calculated as theratio of the force to the distance. If desired, haptic device 113 itselfmay be interfaced with a portion of anatomy 114 and the force determinedbased on the torques provided by the actuators. In such an embodiment,haptic device 113 may make small or large movements or press againstportions of anatomy 114 in an autonomous mode without any physicalassistance from the user. The force may be determined using any Jacobianmethod now known or later developed. The graphical representation 186 ofFIG. 7B illustrates the force with which surgical tool 112 comes incontact with anatomy 114 as a function of displacement of anatomy 114.

If desired, other types of sensing devices may be coupled to hapticdevice 113 or surgical tool 112 to determine other properties of anatomy114. These properties may be used to determine the type of tissue thatis in proximity to haptic device 113. Thus, haptic device 113 may beused to differentiate between hard and soft bones, healthy and diseasestissues, different types of healthy tissues, boundaries of anatomicalstructures, etc. Based on information received from haptic device 113,the type of the tissue may be automatically determined by CAS system 11and displayed on display device 30.

FIG. 9 is a flowchart of a representative method 190 for using hapticdevice 113 as an input device. In step 192, the input mode is initiated.The user may initiate the input mode by any mechanism now known or laterdeveloped. For example, the user may use a graphical user interface, afootswitch, a keyboard, a button, and/or the like, to indicate that theuser desires to use haptic device 113 as an input device. Haptic device113 may control a plurality of objects. However, it is desirable that itonly control a single object at one time. As such, in step 194, anidentification of an object to be controlled is received. The controlledobject may be a cursor, a button, an on-screen dial, a knob, a sliderbar, or other similar user interface object, a virtual implant, asurgical approach trajectory, a bone resection, and/or the like. Theuser may select the object to be controlled by any method now known orlater developed, for example by selecting the object using aconventional input device.

In step 196, a reference pose for haptic device 113 may be stored. Thereference pose is preferably the current pose of haptic device 113. Forexample, in this step, position information about the tip of hapticdevice 113 may be stored. In step 198, the controlled object iscorrelated with haptic device 113. The correlation of the controlledobject with haptic device 113 is desirable so that movement of hapticdevice 113 may be translated or mapped into a corresponding movement oraction relative to the controlled object. The correlation or mappingallows a determination of the amount or direction of movement of thecontrolled object in response to movement of haptic device 113. Forexample, the user may specify that movement of haptic device 113 by oneunit should cause a controlled object, for example cursor 31, to move byten pixels on display device 30.

The user may move haptic device 113 around to control the objectselected in step 194. In step 200, a change in pose of haptic device 113is determined. The change in pose of haptic device 113 is preferablydetermined relative to the reference pose of haptic device 113. Thechange in pose of haptic device 113 may comprise, for example, a changein position of the tip of haptic device 113.

In step 202, the reference pose of haptic device 113 may be updated.Preferably, the reference pose is updated based at least in part on thechange in pose of haptic device 113. If desired, the reference pose maybe updated based at least in part on a wrench applied to haptic deviceby the user. The wrench may be explicitly measured by a sensor. Ifdesired, the wrench may be implicit in that the haptic device candetermine that a wrench is being applied.

In step 204, new parameters for the controlled object are calculated.The parameters of the controlled object may be, for example its pose,position, angle, size, color, shape, orientation, view direction,brightness, contrast, table indices, status, mode, configuration, and/orthe like. The new parameters may be calculated based on the change inpose of haptic device 113 and/or the wrench applied to haptic device bythe user. If desired, the new parameters may be calculated based on thechange in reference pose of haptic device 113. Preferably, correlationinformation obtained in step 198 is used to calculate the newparameters. The new parameters may be used to change the controlledobject. Thus, for example, when the controlled object is cursor 31 andthere is a change in pose of haptic device 113, then a new pose for thecontrolled object may be determined based on the new parameters. In step206, the controlled object is changed based on the new parameters. Thus,for example, if the controlled object is cursor 31, then the position ofcursor 31 on display device 30 may be changed based at least in part onthe new parameters calculated in step 204.

In step 208, a haptic wrench applied by the haptic device to the medicaldevice and/or the user is determined. The haptic wrench may bedetermined based on the new parameters of the controlled object, thechange in pose of haptic device 113, and/or the current pose of hapticdevice 113.

In step 210, the determined haptic wrench is applied to haptic device113. Instead of allowing haptic device 113 to be moved in any direction,it may be desirable to constrain the movement of haptic device 113. Thedetermined haptic wrench when applied to haptic device 113 prevents itfrom moving in certain undesirable directions. For example, if thecontrolled object is capable of moving in only one dimension, it may bedesirable to constrain the motion of haptic device 113 so that hapticdevice 113 moves in only one direction. As another example, when theobject being controlled is cursor 31 on display device 30, then it maybe desirable to constrain the movement of haptic device 113 to atwo-dimensional plane corresponding to display device 30. As a furtherexample, if it is not desirable for haptic device 113 to move largedistances from the reference pose, the haptic wrench may act to returnhaptic device 113 to the reference pose in one or more directions.

Within the input mode, haptic device 113 may be used in a positioncontrol mode or a rate control mode. In the position control mode, thechange in pose of the controlled object tracks the change in pose ofhaptic device 113. For example, if haptic device 113 is moved in aparticular direction by one unit, the controlled object moves in acorresponding direction by a corresponding amount. When haptic device113 is released, it stays in its new pose.

On the other hand, in the rate control mode, the displacement of hapticdevice 113 from the reference pose and/or the wrench applied to thehaptic device by the user, may control the velocity of the controlledobject. For example, if haptic device 113 is maintained in its referencepose (or if no wrench is applied to the haptic device by the user), thenthe rate of movement of the controlled object is zero. The displacementof haptic device 113 from the reference pose (or the magnitude of thewrench applied by the user to the haptic device) determines the velocityof movement of the controlled object with the velocity of movement beingproportional to the displacement of the controlled object (or to themagnitude of the wrench applied to the haptic device). When it isdesirable to move the controlled object, haptic device 113 is simplymoved (or pushed) in the direction of the desired motion of thecontrolled object. When haptic device 113 is released it moves back tothe reference pose due to application, in step 210, of the haptic wrenchdetermined in step 208. Thus, in the rate control mode, the controlledobject may be moved a substantial distance without substantially movinghaptic device 113.

In step 212, a determination is made as to whether haptic device 113 isstill operating in the input mode. If haptic device 113 is not operatingin the input mode, then the process terminates. Otherwise, in step 214,a determination is made as to whether a new object to be controlled hasbeen specified. If a new object to be controlled has not been specifiedthen the process starting at step 200 to determine the change in pose ofhaptic device 113 may be executed. Otherwise, the process starting atstep 194 to receive identification of the new object to be controlled isexecuted.

For example, in one embodiment, the reference pose may be associatedwith the desired trajectory of a drill guide attached to haptic device113. In such an embodiment, updating the reference pose in step 202comprises changing the desired trajectory of the drill guide. When theuser moves haptic device 113 from the reference pose for a prolongedperiod of time, the reference pose will be updated to move in thedirection of the user's deflection. If, in step 210, an appropriatehaptic feedback wrench is applied, then upon release of haptic device113 by the user, haptic device 113 will assume the new reference pose.When the user is satisfied with the reference pose and the input mode isterminated in step 212, haptic device 113 will be in a pose such thatthe drill guide is aligned with the desired trajectory.

The illustrated method for using a haptic device as an input device maybe implemented in software, hardware, or a combination of both softwareand hardware. The steps discussed herein need not be performed in thestated order. Several of the steps could be performed concurrently witheach other. Furthermore, if desired, one or more of the above describedsteps may be optional or may be combined without departing from thescope of the present invention.

A technical advantage of using a haptic device as an input device in themanner described above is that the use of an additional input device maybe avoided thereby reducing the clutter in the operating room.

FIG. 10 illustrates a system for conveying scalar information during amedical, surgical or interventional procedure. Haptic device 113 may bedeployed as a fully integrated component of CAS system 11 or act as anoptional peripheral to such a system. Location or position informationof tool 112 coupled to haptic device 113 may be sensed and provided backto CAS system 11 with or without the use of sensor(s) 14.

Medical, surgical and interventional procedures will be referred tocollectively as “medical procedures.” The medical procedure may involveresecting a portion of an anatomy, such as for a joint replacement,joint resurfacing, tumor removal, bone deformity correction and/or thelike. If desired, the medical procedure may involve applying asynthetic, biologic, or therapeutic substance to a surface or region ofinterest, or placing a sensor, probe, implant or radioactive material ina desired position, surface or volume. When a user interacts, forexample, with a haptic object, it is sometimes desirable to know themagnitude of forces applied to the haptic object or the amount that areal or virtual tool or implant is penetrating the haptic object. Fornon-trivial haptic objects, or those with complicated two or threedimensional forms, it may be difficult to present this information in amanner that is simple for the user to understand. The desirable piece ofinformation is often the local distance to a surface of interest or to adesired position, the local penetration distance of the surface ofinterest, or haptic repulsion force. While these can be up tothree-dimensional vector quantities, the magnitude (or length) of suchvectors, possibly in the direction of a local unit normal to the hapticobject, are most useful for augmenting the haptic interaction of theuser. These magnitudes are simple one-dimensional quantities and can beconveyed to the user in a variety of methods, including meters, dials,numerical displays, graphs, and other visual methods, but also withaudio, tactile, haptic, or other means.

In the exemplary embodiment of FIG. 10, CAS system 11 is communicativelycoupled to an audio source 216, for example a speaker, and displaydevice 30. During an exemplary medical procedure, anatomy 114, which inthe embodiment of FIG. 10 is a bone, is to be cut along haptic object 20using the surface of haptic object 20 as a guide surface. During themedical procedure, one-dimensional information, for example informationregarding the scalar distance D of the tip of tool 112 from the surfaceof an object of interest, for example haptic object 20, is provided tothe user, preferably but not necessarily automatically. CAS system 11may be programmed, for example, to automatically provide this scalarinformation based on the stage of the procedure, the type of tool beingused, the tool's position or proximity to an object (including surface)of interest, or other cues that assist CAS system 11 in identifying theobject with respect to which magnitude information will be determinedand displayed.

In an exemplary embodiment, prior to initiation of the cuttingoperation, the value of D is positive. A value of zero for D indicatesthat the tip of cutting tool 112 is at the desired depth inside anatomy114. In the exemplary embodiment, the desired depth is at the surface ofhaptic object 20. A negative value for D indicates that the depth of thetip of cutting tool 112 inside the bone is more than that desired.One-dimensional information may be communicated to the surgeon by any ofa variety of methods, such as visual, audio, tactile, haptic, and/or thelike. For example, a visual indicator 218, for example a level meter, adial, numerical displays, graphs, etc., may be used to indicate thevalue of D on display device 30 or any other device in proximity to theuser. If desired, audio source 216, visual indicator 218 and/or displaydevice 30 may be provided closer to the user, for example on tool 112,haptic arm 113, other handheld tools, instruments or accessories, orwearable visual, audio, haptic, or tactile displays. For example, tool112 may include a simple display device or a multi-colored indicator,for example a multi-colored LED indicator, a multi-colored lampindicator, a LED level meter, and/or the like, to indicate the cuttingdepth to the user. In such an embodiment, the user does not need to taketheir attention away from the surgical working area. Similarly, tool 112may include an audio source to indicate the cutting depth to the user.In such an embodiment, the audio indication from audio source 216 may beeasily heard by the user because of its proximity to the user.

If desired, audio indicators or tones from speaker 216 may be providedinstead of or in addition to visual indicator 218. For example, a seriesof beeps may be provided to indicate the value of D. As the value of Ddecreases, the interval between the beeps may be reducedcorrespondingly. If desired, the beeps may turn into a buzzing or anyother sound when the value of D becomes zero and into a different sound,for example, a high pitched sound when the value of D becomes negative.In an exemplary embodiment, a positive value for D is an acceptablevalue and a negative value for D is an unacceptable value.

An advantage of providing an audio indicator is that the surgeon doesnot have to take his/her eyes off the patient's anatomy, such as bone114. However, if the noise in the operating room makes it difficult forthe surgeon to hear an audio indicator, then a visual indicator may bemore appropriate.

If a haptic system or device is used, such as shown in the illustratedexemplary embodiment, a tactile indication may be provided throughhaptic arm 113 or through an auxiliary device. In an embodiment, hapticarm 113 vibrates to provide tactile indication to the user. Thefrequency, amplitude, waveform, and/or other property of vibration ofhaptic arm 113 is dependent on distance D. In another embodiment, avibrating device may be provided on the user's body. The frequency,amplitude, waveform, and/or other property of vibration of the vibratingdevice is dependent on distance D. An advantage of providing a tactileindication is that the surgeon does not have to take his/her eyes offbone 114.

Because the one-dimensional information is easy to communicate to theuser, the user is able to focus his attention on the task of cuttingknowing that an audio indication, a tactile indication or a quick glanceat visual indicator 218 will inform him if he is cutting to theappropriate depth.

Depth information may also be displayed to the user in situations wherethe user is not using a haptic device. For example, the user may becutting the bone freehand with a tool whose pose is tracked by atracking system. In such an embodiment, the user does not have thebenefit of the haptic feedback that will tend to apply forces to thetool that keep it on the surface of a haptic object or prevent it frompenetrating a haptic object. Instead of a haptic object, a simplegeometric object, e.g., a curve, a point, line, a surface or a volume,may be used as the object of interest. The object of interest may beidentified to the CAS system or the CAS system may determine it based onother information it has. For example, the object of interest may bedefined directly with respect to the patient or with respect to adiagnostic image of the patient, or it can be derived or determined bythe CAS system from other anatomical points or features identifiable bya user. The CAS system knows the position of the tool relative to theobject of interest and displays one-dimensional depth information. Insuch a system, D may be calculated by determining the distance from thetip of tool 112 to the desired depth of cut of the object of interest.If desired, the object of interest may comprise a curve, a point, asurface, a line, a volume, or a set of desired positions. The object ofinterest may be a haptic object, a geometric object and/or the desiredshape of the portion of the anatomy.

If desired, in an alternative embodiment, the one dimensional depthdisplay may be augmented with two-dimensional representations of thecutting depth at each point on the surface or three-dimensionalrepresentations of each point in space. The two-dimensional orthree-dimensional representation may be a schematic or realisticdepiction of the surface of interest. A property, for example color,brightness, and/or the like, of points of the surface of interest may bebased at least in part on the distance between the tool and therespective points when the tool was closest to the respective pointsduring the medical procedure. For example, different colors may be usedto denote the distance between the tool and the points. The position ofthe tool when it was closest to a point during the medical procedure isdetermined and the distance between the tool and the point calculated.The color of the point reflects the distance from the tool to the pointwhen it was closest to that point.

FIG. 11 is a flowchart of a method 220 for conveying depth informationduring a medical procedure. Method 220 is preferably executedperiodically. In step 222, the distance of tool 112 from the desiredsurface is determined. Preferably, the distance D of the tip of tool 112from the surface of an object, such as haptic object 20, is determined.The process for determining a value for D depends upon how the object isrepresented internally. This representation of the object may be basedon surface polygons, voxels, NURBs, constructive solid geometry, and/orany other method for representing geometrical objects now known or laterdeveloped. The distance from the current location of tool 112 to anappropriate point on the surface of haptic object 20 is calculated. Thecalculated distance is assigned a positive or negative value dependingon the relative position of the tip of tool 112 and bone 114 withrespect to the surface of haptic object 20. In the example illustratedin FIG. 10, if the tip of tool 112 and bone 114 are on opposite sides ofthe surface of haptic object 20, then D is assigned a positive value.Otherwise, D is assigned a negative value.

In step 224, the determined distance is mapped to a desired outputformat for display to the user. For example, the distance may be mappedto an appropriate color, audio frequency, time period, sound, image,haptic cue, and/or the like. Preferably, the mapping is performed basedat least in part on the value of D. Table A below shows an exemplarymapping table for a system where visual signals are provided to theuser.

TABLE A D (Millimeter) OUTPUT FORMAT 1-2  Green light 0.1-0.99 YellowLight 0.0-0.09 Red Light <0.0 Black Light

In step 226, the determined distance is conveyed to the user in thedesired output format. If the desired output format is a visualindicator, then in an exemplary embodiment an indicator of theappropriate color is displayed. For example, as specified in exemplaryTable A, if the value of D is within an acceptable range, say between 1and 2 millimeters, then a green indicator is displayed, if the value ofD is between 0.1 and 0.99 millimeters, then a yellow indicator isdisplayed, if the value of D is between 0.0 and 0.09 millimeters, then ared indicator is displayed and if the value of D is in an unacceptablerange, say less than zero, then a black indicator is displayed. In theembodiment of FIG. 10, information about distance D is displayed bylevel meter 218. As the value of D changes, the color of a level inlevel meter 218 is changed. If the desired output format is an audioindicator, then in an exemplary embodiment, distance information isconveyed by beeps as discussed herein with reference to FIG. 10.

A technical advantage of an exemplary embodiment of the presentinvention is that during a medical procedure information about the depthof a tool may be provided to the user in a simple manner so that theuser may focus his/her attention on the medical procedure.

1. (canceled)
 2. A teleoperation system for use during a surgicalprocedure, said teleoperation system comprising: a surgical tool; amaster input device to be moved by a user; a slave device coupled tosaid surgical tool and responsive to movement of said master inputdevice, said slave device configured to be operated in a first mode toplace said surgical tool in a desired pose with respect to a targetregion and a second mode to manipulate said surgical tool during thesurgical procedure; and a visual indicator coupled to said slave deviceand configured to emit light in said second mode to provide visualinformation to the user in said second mode.
 3. The teleoperation systemof claim 2, wherein said visual indicator is configured to emit light ina plurality of different colors in said second mode to provide visualinformation to the user in said second mode.
 4. The teleoperation systemof claim 3, wherein said visual indicator comprises a multi-colored LEDindicator.
 5. The teleoperation system of claim 4, further comprising adisplay configured to display said surgical tool.
 6. The teleoperationsystem of claim 2, further comprising a control system configured tolock said slave device in a desired pose.
 7. The teleoperation system ofclaim 6, wherein said control system is configured to adjust saiddesired pose of said slave device in response to receiving a manualforce on said slave device.
 8. The teleoperation system of claim 6,wherein said control system is configured to lock only a portion of saidslave device so that the user can adjust said slave device.
 9. Theteleoperation system of claim 2, further comprising a control systemconfigured to lock a trajectory of a tool guide passing through thetarget region.
 10. The teleoperation system of claim 9, wherein saidcontrol system is configured to allow the user to pivot said tool guideabout a pivot point while said trajectory is locked.
 11. A teleoperationsystem for use during a surgical procedure, said teleoperation systemcomprising: a surgical tool; a master input device to be moved by auser; a slave device coupled to said surgical tool and responsive tomovement of said master input device, said slave device configured to beoperated in a first mode to place said surgical tool in a desired posewith respect to a target region and a second mode to manipulate saidsurgical tool during the surgical procedure; and a visual indicatorcoupled to said surgical tool and configured to emit light in saidsecond mode to provide visual information to the user in said secondmode.
 12. The teleoperation system of claim 11, wherein said visualindicator is configured to emit light in a plurality of different colorsin said second mode to provide visual information to the user in saidsecond mode.
 13. The teleoperation system of claim 12, wherein saidvisual indicator comprises a multi-colored LED indicator.
 14. Theteleoperation system of claim 13, further comprising a displayconfigured to display said surgical tool.
 15. The teleoperation systemof claim 11, further comprising a control system configured to lock saidslave device in a desired pose.
 16. The teleoperation system of claim15, wherein said control system is configured to adjust said desiredpose of said slave device in response to receiving a manual force onsaid slave device.
 17. The teleoperation system of claim 15, whereinsaid control system is configured to lock only a portion of said slavedevice so that the user can adjust said slave device.
 18. Theteleoperation system of claim 11, further comprising a control systemconfigured to lock a trajectory of a tool guide passing through thetarget region.
 19. The teleoperation system of claim 18, wherein saidcontrol system is configured to allow the user to pivot said tool guideabout a pivot point while said trajectory is locked.
 20. A method forusing a teleoperation system during a surgical procedure, theteleoperation system comprising a surgical tool, a master input deviceto be moved by a user, a slave device coupled to the surgical tool, anda visual indicator, the slave device operable in a first mode to placethe surgical tool in a desired pose with respect to a target region inresponse to a manual force applied to the slave device, said methodcomprising the steps of: controlling the slave device in a second mode,different than the first mode, to manipulate the surgical tool duringthe surgical procedure in response to movement of the master inputdevice by the user; and visually indicating information to the user inthe second mode by emitting light with the visual indicator in thesecond mode.
 21. The method of claim 20, further comprising emittinglight in a plurality of different colors in the second mode to providevisual information to the user in the second mode.
 22. The method ofclaim 21, wherein the visual indicator comprises a multi-colored LEDindicator.
 23. The method of claim 22, further comprising displaying thesurgical tool on a display.
 24. The method of claim 20, furthercomprising locking the slave device in a desired pose.
 25. The method ofclaim 24, wherein locking the slave device in the desired pose isfurther defined as locking only a portion of the slave device so thatthe user can adjust the slave device.
 26. The method of claim 20,further comprising locking a trajectory of a tool guide passing throughthe target region.
 27. The method of claim 26, further comprisingenabling the tool guide to be pivoted about a pivot point while thetrajectory is locked.